feat(destroy): graceful SIGINT handling — release lock + preserve state on interrupt - #826
Merged
Merged
Conversation
…n first Ctrl-C) Closes #816 (deferred "optional fix 3" from #804 / PR #814). Before this change `cdkd destroy` / `cdkd state destroy` had no SIGINT handler, so a first Ctrl-C killed the process mid-destroy: the `finally` that releases the stack lock never ran (lock stranded for its full TTL), and any in-flight provider delete was severed abruptly. The destroy runner now registers a per-call SIGINT handler (Terraform parity): - First Ctrl-C sets a `draining` flag. The reverse-DAG delete loop checks it before scheduling each subsequent LEVEL (and, defense-in-depth, before dispatching each resource), so no new delete starts; the deletes already in flight in the current level are awaited to completion (not cancelled). Control then falls through to the existing `finally`, which flushes the incremental save-chain from #804 (preserved state lists only resources that still exist), stops the live renderer, and releases the lock. The destroy exits non-zero. - Second Ctrl-C bypasses graceful shutdown (process.exit(130)). On a graceful interrupt the runner PRESERVES state (does not deleteState even though errorCount is 0, because resources remain) and surfaces a new `DestroyRunnerResult.interrupted` flag; both destroy.ts and state.ts stop their multi-stack loop on the first interrupted stack and throw PartialFailureError (exit 2). The handler reads/writes only its own call's closure state and is removed via process.removeListener in the `finally`, so no listener leaks. Nested- stack recursion registers one handler per level; Node delivers SIGINT to every listener, so the first Ctrl-C drains the parent and every in-flight child. Tests: 5 unit tests in destroy-runner-sigint.test.ts (the handler is captured by spying on process.on('SIGINT', ...) and invoked directly — no real OS signal is sent). Happy-path destroy is unchanged. Docs: destroy-interruption subsection in docs/state-management.md + stale-lock note in docs/troubleshooting.md + changelog entry.
Fix one blocker plus two smaller findings from the independent review of the #816 graceful-SIGINT implementation. BLOCKER: a second Ctrl-C called process.exit(130) synchronously, bypassing the finally that releases the stack lock — re-introducing the 30m-stranded- lock bug on the force-quit path. The second-Ctrl-C handler now fires a best-effort un-awaited releaseLock() AND always prints the exact recovery command to stderr ("Force-quit: stack lock may not be released. If the next run reports a lock, run: cdkd force-unlock <stackName>") before exiting, so a stranded lock is always recoverable deterministically. Updated the unit test to assert both the recovery message (with the real stack name) and the best-effort release attempt. Documented the final force-quit semantics in docs/troubleshooting.md. Minor: deep nesting + high concurrency register many process SIGINT listeners (one per nested level, plus per-provider handlers) and can exceed Node's default 10-listener cap, emitting a misleading MaxListenersExceededWarning. Raise the cap to 100 (via Math.max so recursion never lowers it) with a comment explaining the headroom — leaves the warning active above 100 so a real leak is still surfaced. Nit: state.ts only had an inner per-region-loop break on interrupt and relied on re-entry to stop the outer stack loop. Added an explicit outer-loop guard mirroring destroy.ts's stack-loop break.
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 13, 2026
# [0.221.0](v0.220.5...v0.221.0) (2026-06-13) ### Features * **destroy:** graceful SIGINT handling — release lock + preserve state on interrupt ([#826](#826)) ([06d1c8b](06d1c8b))
|
🎉 This PR is included in version 0.221.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cdkd destroyhad no SIGINT handler. A Ctrl-C killed the process mid-destroy: the stack lock was left behind (TTL 30m) and thefinallycleanup never ran. This adds graceful SIGINT (Terraform parity), building on #804's incremental destroy persistence.Behavior
breaks at the level boundary; per-resource defense-in-depth gate), let in-flight provider deletes finish (await Promise.all), then fall through tofinallywhich flushes the incremental save-chain (destroy: interrupted/partially-failed destroy replays Custom Resource delete against an already-deleted backing Lambda — 10-minute stall on re-run #804) + releases the lock. State is preserved on interrupt (preserveState = errorCount > 0 || interrupted— nodeleteStateeven at 0 errors, since resources remain), so a re-run resumes cleanly with no replay and no 30m lock wait.releaseLock+ always printsForce-quit: stack lock may not be released ... run: cdkd force-unlock <stack>to stderr, thenprocess.exit(130).DestroyRunnerResult.interrupted;destroy.ts+state.tsbreak their multi-stack loops on the first interrupted stack and exit non-zero.finally(no leak across stacks / nested-stack recursion).process.setMaxListenersbumped (Math.max) so deep nesting + per-provider SIGINT handlers don't trip a spurious MaxListenersWarning.Test plan
tests/unit/cli/destroy-runner-sigint.test.ts, 5): first Ctrl-C finishes in-flight + schedules no new + preserves trimmed state + releases lock + marks interrupted; level-boundary gate (diamond DAG); second Ctrl-C → best-effort release + recovery message + exit(130); normal completion → not interrupted + listener removed; removeListener in finally. Full suite 5740 tests pass.microservicesbroad integ): deploy + destroy clean — 19 deleted, 0 errors, 0 orphans — confirms the SIGINT handler leaves the happy-path destroy unchanged.Independent review
Code review of the implementation + a focused re-review of the blocker fix-back: both clean, no blockers. The original review's BLOCKER (2nd-Ctrl-C stranded the lock), MaxListeners minor, and state.ts break-placement nit were all addressed and verified.
Deferred / related
The IGW/NAT delete-ordering gap (the other half of the original #804 incident) shipped separately as #823 (#817).
Closes #816